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I.   INTRODUCTION 


A.  PROJECT  HISTORY 

Evaluation  as  a  management  support  function  was  institutionalized  at  the 
National  Endowment  for  the  Arts  in  October  1971  with  the  establishment  of  the 
Endowment's  Evaluation  Office.  By  1979  three  types  of  evaluation  were  in 
place.  "In  the  past,  evaluation  has  been  used  largely  to  pre-screen  appli- 
cants so  that  panels  can  make  grant  recommendations  based  on  the  applicant's 
quality  and  potential  ability  to  achieve  the  objectives  specified  in  the 
guidelines.  The  second  most  frequent  type  of  evaluation  undertaken  has  been 
an  assessment  of  the  performance  of  the  grantee  after  receipt  of  the  grant 
award.  This  is  known  as  'grantee  specific'  evaluation.  This  gives  the  Endow- 
ment an  idea  of  how  effectively  the  grantee  is  using  the  grant.  The  third 
type  of  evaluation  is  done  to  measure  program  category  effectiveness.  In 
other  words,  it  provides  information  about  whether  the  funding  category  is 
actually  meeting  its  objectives  and  contributing  to  the  attainment  of  the 
Endowment's  overall  goals."  Also  in  1979,  it  was  anticipated  that  "As 
operational  planning  is  undertaken,  the  programs  will  be  identifying  measur- 
able and/or  observable  objectives.  Once  done,  evaluation  can  be  pegged  to 
these.  Assessment  methods  appropriate  to  arts  support  can  be  improved  as  a 
result."  2 

The  Office  of  Evaluation  operational ized  the  Endowment's  position  that  "Evalu- 
ation, therefore,  is  important  both  as  an  implementation  tool  and  as  an  aid  to 

3 
planning."    By  1980,  however,  the  evaluation  function  had  been  subsumed, 

with  decreased  emphasis,  under  the  overall  responsibilities  of  the  Research 
Division. 


1  General  Plan,  1980-1984.   National  Endowment  for  the  Arts,  Office  of 
Policy  and  Planning,  April  1979,  page  146. 

2  Ibid,  page  147 

3  Ibid,  page  147 
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With  a  change  in  leadership,  the  Endowment  responded  to  the  New  Federalism  by 
placing  increased  emphasis  on  improved  management  procedures  and  accountabil- 
ity. Policy  changes  were  implemented  to  reflect  this  new  emphasis;  for  ex- 
ample, the  submission  of  Final  Descriptive  and  Statistical  Reports  are  now  a 
formal  prerequisite  for  consideration  for  award  of  subsequent  grants.  In 
1982,  the  Research  Division  issued  Program  Solicitation  82-1,  Technical  Assis- 
tance for  a  Pilot  Program  of  Evaluation  Studies,  which  acknowledged  the  En- 
dowment's intention  "to  resume,  by  means  of  this  pilot  effort,  the  support  of 
program  evaluation  studies...  It  is  expected  that  these  studies  will  provide 
the  necessary  experience  to  develop  program  evaluation  studies  into  a  contin- 
uing activity." 

Three  Endowment  offices  volunteered  to  participate  in  the  pilot  study  and 
identified  target  subject  areas: 

o  Office  of  the  Deputy  Chairman  for  Management:  Analysis  and  Use  of 
Final  Descriptive  Reports  from  Grantees 

o  Literature  Program:  Literary  Magazines  and  Small  Presses  Category 

o  Design  Arts  Program:  Design  Demonstration  Category 

Evaluation  Technologies  Incorporated  (ETI)  was  awarded  a  contract  to  provide 
evaluation  technical  assistance  in  March  1983. 

B.  PURPOSE  OF  THIS  REPORT 

The  primary  purposes  of  this  report  are  to  chronicle  the  evaluation  design  and 
implementation  activities  performed  by  ETI  over  the  course  of  two  contracts 
and  the  periods  March  1983  -  August  1984  and  June  1985  -  May  1986  ;  to  describe 
the  evaluation  processes  applied;  and  to  summarize  the  effects  of  the  effort 
and  to  offer  ETI ' s  insights  on  the  potential  for  evaluation  applications  at 
the  Endowment. 
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II.  CONTRACT  REQUIREMENTS 


A.  ENDOWMENT  GOALS 

The  implicit  goals  of  the  contracted  assignment  were  to  test  the  application 
of  evaluation,  e.g., 

o  Is  evaluation  feasible  for  the  Endowment  given  the  highly  subjec- 
tive and  non-quantifiable  nature  of  art? 

o  Can  program  activities  be  defined  within  an  evaluation  framework 
without  infringing  upon  or  threatening  subjective  and  expert  panel 
judgements? 

o  Can  evaluation  activities  be  performed  by  Endowment  staff  within 
the  context  of  their  current  grant-cycle  responsibilities? 

o  Can  evaluation  results  be  a  useful  tool  for  managers  and  panels? 

The  intent  of  this  report  is  to  support  a  resounding  yes  to  each  of  these 
questions. 

B.  CONTRACT  OBJECTIVES 

The  initial  contract  effort  was  directed  toward  defining  and  planning  evalua- 
tion studies,  to,  through  conferences  with  program  staff,  assess  specific 
needs  and  offer  appropriate  methods  for  the  formulation  of  approaches  tailored 
to  meet  those  needs.  ETI  was  tasked  with  the  following: 

o  Set  appropriate  quantifiable  goals 

o  Find  methods  to  economically  collect  data  to  measure  achievements 
toward  reaching  those  goals 
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o  Develop  procedures  for  analyzing  data  and  integrating  the  results 
into  the  decision-making  process 

o  Present  project  results  for  agency  implementation. 

ETI  understood  that,  in  addition  to  developing  program  evaluation  study  de- 
signs, we  were  to  work  closely  with  program  staff  members  throughout  the  pro- 
cess so  as  to  facilitate  an  internal  staff  evaluation  design  capability.  That 
is,  to  show  Endowment  staff  how  to  design  evaluation  studies  through  demon- 
stration and  encouraging  their  participation  throughout  the  process.  Further- 
more, ETI  was  to  design  the  studies  and  provide  guidance  on  their  implemen- 
tation so  as  to  allow  the  research  and  analysis  to  be  performed  by  in-house 
staff. 

Following  a  one-year  period  of  no  internal  action  on  the  evaluation  plans  pre- 
pared by  ETI,  ETI  was  again  contracted  to  implement  the  evaluation  studies  for 
each  program,  with  their  assistance  in  the  collection  of  grantee  data.  ETI 
was  requested  to  collect,  process,  and  analyze  evaluation  data,  and  provide 
written  reports  on  findings.  ETI,  in  the  process,  also  established  automated 
data  bases  for  the  programs  and  made  recommendations  to  the  programs  regarding 
revised  program  and  grantee  data  collection  schemes  which  would  enhance  future 
evaluation  efforts.  The  automated  systems  were  prepared  so  as  to  allow  con- 
tinued, expanded  use  by  program  staff.  ETI,  in  essence,  had  developed  histo- 
rical data  bases  which  allowed  for  easy  updates  as  grant  awards  are  made  each 
year,  and  thus  created  the  internal  ability  to  have  on-going  program  evalua- 
tion. 

C.  PROGRAM  PARTICIPANTS 

Initially,  three  Endowment  offices  volunteered  to  participate  in  the  project: 

o  Office  of  the  Deputy  Chairman  for  Management 
o  Literature  Program 
o  Design  Arts  Program. 


Later  in  the  contract  period,  a  fourth  office  was  included.  The  Inter-Arts 
Program  was  substituted  when  the  Deputy  Chairman  for  Management  left  the 
agency. 


III.  EVALUATION  DESIGN 


A.  GOAL-BASED  EVALUATION  DESIGN  PROCESS 

The  procedure  ETI  followed  in  defining  evaluation  objectives  and  developing 
the  evaluation  plans  for  each  office  was  fairly  standard,  and  is  summarized 
below: 

o  Review  of  program  literature  regarding  current  activities,  pre- 
vious studies,  program  documentation,  and  grant  application 
guidelines 

o  Repeated  conferences  with  program  staff  to  discuss  program/ 
category  activities  and  possible  evaluation  issues  for  study,  to 
define  evaluation  objectives,  and  to  identify  existing  and  poten- 
tial data  sources 

o  Preparation  of  draft  and  final  goal  statement  matrices ;  staff 
reviews  and  input 

o  Preparation  of  evaluation  strategy  papers  ;  staff  reviews  and  input 

o  Development  and  staff  reviews  of  draft  and  final  evaluation  plans 

o  Submission  of  detailed  work  plans,  instructing  program  personnel 
on  the  implementation  of  the  evaluation  plans. 

ETI  began  its  work  with  each  program  by  conducting  a  series  of  conferences 
with  Endowment  staff  in  each  assigned  office.  Our  first  task  was  to  discuss 
the  differences  between  evaluation  and  research.  We  then,  together,  explored 
various  evaluation  issues  of  importance  to  the  planning  and  development  of  the 
office,  including  how  the  evaluation  findings  would  be  used  and  by  whom. 

With  a  clear  definition  of  an  evaluation  question,  ETI  then  prepared  a  goals 
matrix.   The  purpose  of  the  matrix  was  to  visually  portray  the  relationships 
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between  Endowment,  program,  and  category  goals  and  to  further  illustrate  how 
the  program  category  activities  can  be  defined  for  evaluation  purposes.  The 
matrices  contained  the  following: 

o  Endowment  goals 

o  Program  goals 

o  Category  goals 

o  Goal  appraisal  factors 

o  Indicators/measures  (pre-grant  and  post-grant) 

o  Data  sources 

o  Data  analysis  plans 

o  Hypotheses  and  assumptions 

Upon  program  staff  approval  of  the  matrix,  ETI  prepared  an  evaluation  strategy 
paper  which  further  defined  the  proposed  evaluation  framework  by  discussing: 

o  Evaluation  focus,  the  category  goals  and  evaluation  objectives  to 
be  addressed 

o  Uses  of  evaluation  information,  identification  of  the  audiences 
and  uses  of  evaluation  outcomes 

o  Plans  for  the  evaluation  design,  description  of  the  goal -based 
approach. 

The  evaluation  strategy  paper  outlined  the  type  of  evaluation  (i.e.,  formative 
or  summative)  to  be  designed,  the  types  of  information  to  be  generated  (e.g., 
project  achievement  at  the  individual  grantee  level) ,  the  measurement  points 
(e.g.,  comparison  of  pre-  and  post-grant  indicators  for  each  project),  and  the 
anticipated  uses  of  the  evaluation  findings  (e.g.,  as  input  for  short-range 
program  management  decisions  and  panel/funding/policy  decisions;  program 
advocacy!  . 

Upon  staff  review  and  concurrence  with  the  proposed  evaluation  strategy,  ETI 
proceeded  to  prepare  the  evaluation  plan.  The  evaluation  plans  included  the 
following  basic  information: 

7 


o  Background  information  on  the  identification  and  definition  of  the 
evaluation  objectives 

o  An  overview  of  the  evaluation  framework  and  methodology 

o  Detailed  data  collection  and  analysis  plans  for  each  objective 

o  An  outline  for  preparation  of  a  final  evaluation  report. 

ETI  also  prepared  separate  work  plans  as  an  accompaniment  to  the  evaluation 
plan.  The  work  plan  described  the  nature,  scope,  and  sequence  of  tasks  in- 
volved in  the  implementation  of  the  evaluation  plan,  including  step-by-step 
procedures  and  potential  pitfalls. 

B.  DESIGN  PROCESS  OUTCOMES 

ETI's  work  with  each  of  the  participating  offices  was  thoroughly  documented, 
with  each  office  receiving  no  less  than  six  complete  documents  as  described 
above.  In  addition  to  the  production  of  those  materials,  and  the  individu- 
alized technical  assistance  provided  throughout  their  development,  certain 
other  benefits  were  realized.  ETI  concluded  its  work  with  the  following 
insights: 

o  It  was  demonstrated  to  each  of  the  participating  offices  that 
program  activity  which  is  often  considered  "artistic  and  non- 
quantifiable"  can,  in  fact,  be  defined  within  an  evaluation 
framework  without  infringing  upon  or  threatening  subjective  and 
expert  panel  judgements. 

o  Participating  programs  found  the  evaluation  design  process  partic- 
ularly useful  in  defining  program  information  needs  and  purposes, 
and  in  identifying  information  sources  and  gaps. 

o  Participating  programs  suggested  that  evaluation  data  will  benefit 
their  programs  in  a  number  of  ways,  including: 
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--  Program  management:  monitoring  program  activities,  developing 
funding  priorities,  providing  guidance  to  applicants  and 
grantees 

--  Assistance  to  panels:  providing  information  on  the  state-of- 
the-field,  information  on  indicators  of  success,  assistance  in 
determining  funding  priorities  and  performance  standards, 
assistance  in  reviews  of  program  goals,  policies,  etc. 

—  Advocacy:  identification  of  trends  in  the  field  and  of  out- 
standing projects,  and  general  information  on  how  the  category 
is  doing  overall. 

o  Program  and  category  goals  are  frequently  not  expressed  in  measur- 
able terms,  may  not  be  applied  during  the  application  review  pro- 
cess, and  may  not  be  related  to  funding  priorities.  Related  per- 
formance expectations  or  standards  have  not  been  consistently 
established. 

For  example,  ETI  found  that,  frequently,  Endowment  program  goal 
statements  incorporate  words  such  as  "innovative"  and  "of  highest 
quality."  For  evaluation  purposes,  terms  such  as  these  must 
either  be  reworded  or  defined  by  quantitative  performance  indi- 
cators to  allow  meaningful  measurement  of  goal  achievement.  As  a 
case  in  point,  the  Services  to  the  Arts  Category  of  the  Inter-Arts 
Program  expressed  one  objective  in  terms  of  providing  innovative 
business  practices  to  artists  and  arts  organizations.  "Innova- 
tive" was  defined,  for  evaluation  planning  purposes,  in  terms  of 
the  accessibility  of  the  service  to  the  arts  community,  its  re- 
duced cost  to  arts  users,  and/or  its  primary  focus  on  the  unique 
characteristics  and  needs  of  the  arts  users. 

o  Currently,  there  is  little  Endowment-provided  incentive  for  pro- 
grams to  evaluate  goal  achievement. 


That  is,  there  currently  exists  no  Endowment-wide  policy  regarding 
program  evaluation.  Existing  evaluation  efforts  reflect  individ- 
ual program  and  even  personal  desires,  efforts,  and  needs  for 
evaluative  feedback  and  information.  And,  in  fact,  no  formal  in- 
centive exists  to  examine  programs'  histories  of  achievements  when 
planning  for  future  thrusts  and  activities,  as  for  example  in  pre- 
paring the  Congressionally-requested  five-year  plan. 
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IV.   IMPLEMENTATION  OF  EVALUATION  STUDIES 


A.  THE  IMPLEMENTATION  PROCESS 

The  procedure  used  to  implement  the  evaluation  design  was  relatively  straight- 
forward. It  entailed  the  development  of  data  recording  sheets;  data  collec- 
tion, including  reviews  of  the  grant  files,  meetings  with  Endowment  personnel, 
and  telephone  follow-ups  for  missing  information;  data  analysis;  and  finally, 
report  preparation. 

The  development  of  the  data  recording  sheets  was  based  on  the  goals  statement 
matrix  which  had  been  generated  during  the  evaluation  design  process.  The 
data  recording  sheets  were  developed  in  a  spreadsheet  format  to  facilitate 
data  entry  and  subsequent  computer  analysis,  and  included  both  pre-grant  and 
post-grant  information. 

Data  collection  was  a  lengthy  process.  Endowment  programs  maintain  extensive 
files  on  their  grantees,  which  presuppose  a  well-developed  vocabulary  of  terms 
and  usages  particular  to  the  disciplines  and  the  specific  Endowment  programs. 
Much  of  the  information  contained  in  the  files  was  supplementary  to  the  pur- 
poses of  the  evaluation.  There  was  a  pronounced  learning  curve  with  each  of 
the  program's  files,  to  comprehend  the  arrangement  and  composition  of  the 
files,  the  language  usage,  as  well  as  the  location  (or  probable  location)  of 
information  to  be  compiled  on  the  data  recording  sheets. 

The  grant  files  for  one  program  represented  unique  events  and  were  grouped  by 
chronological  year.  The  files  for  another  program  could  also  have  been 
grouped  chronologically,  but,  because  most  of  these  grantees  represented 
organizations  which  had  been  funded  for  several  years  successively,  were 
instead  arranged  by  those  funded  organizations.  This  latter  arrangement 
permitted  a  somewhat  more  historical  approach  to  be  taken. 

As  the  files  were  reviewed,  it  became  apparent  that  the  comparison  built  into 
the  evaluation  design  (i.e.,  that  of  pre-grant  and  post-grant)  could  not  be 
made  with  the  information  contained  in  the  grant  files.   Either  the  dissimi- 
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larities  between  the  Endowment's  pre-grant  and  post-grant  information  require- 
ments were  too  great,  or  else  the  evaluation  design  posed  specific  questions/ 
concerns  which  had  not  previously  been  addressed  by  the  programs  evaluated. 

To  offset  these  gaps  in  information  found  following  the  review  of  the  files, 
telephone  interviews  with  grantees  were  undertaken  to  collect  missing  infor- 
mation. It  was  decided  that  the  likelihood  of  response  would  be  greater  if 
these  were  either  conducted,  or  at  least  initiated,  by  Endowment  personnel, 
rather  than  the  contractor. 

Many  of  the  grantees  contacted  used  this  opportunity  to  voice  concerns  about 
Endowment  procedures,  such  as  the  time  lag  between  applying  for  the  grant  and 
being  awarded  one.  This  adds  to  the  intangibility  of  planning  for  arts  ser- 
vice organizations,  as  funding  situations  may  change  drastically  over  the 
intervening  period. 

Perhaps  the  most  revealing  aspect  of  the  entire  implementation  process  for  the 
data  collection  phase  was  the  scope  and  magnitude  of  the  files,  and  the  corre- 
sponding magnitude  of  the  learning  curve. 

The  data  collection  efforts  were  significantly  enhanced  by  the  extensive  input 
and  assistance  from  the  Endowment  personnel  assigned  to  the  evaluation  and 
also  from  their  colleagues.  For  example,  for  the  Design  Arts  evaluation, 
information  collection  took  place  primarily  at  the  Endowment:  space  was  pro- 
vided for  the  evaluator,  as  were  various  support  services.  A  program  profes- 
sional was  directly  assigned  to  the  evaluation  effort  and  undertook  all  of  the 
telephone  interviews  with  the  grantees  in  order  to  collect  missing  informa- 
tion. Close  coordination  with  regards  to  the  types  of  data  needed  forestalled 
much  confusion  as  to  the  specifics  requested.  In  addition,  this  individual 
was  dedicated  to  the  evaluation  effort,  having  been  hired  in  support  of  this 
project.  This  luxury  facilitated  a  close  collaboration,  whereby  questions  on 
the  references  used  in  data  collection  could 'be  clarified  by  the  contractor, 
and  questions  on  outcomes  and  procedures  used  in  competitions  could  be  readily 
explained  by  that  individual. 
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The  analysis  plans  were  refined  during  the  course  of  the  implementation.  Not 
only  were  significant  numbers  of  responses  missing,  but  program  priorities 
(with  regards  to  the  some  of  the  proposed  analyses)  also  shifted.  Realign- 
ments were  therefore  necessary  and  consisted,  for  the  most  part,  of  excising 
most  of  the  comparative  studies  between  pre-grant  and  post-grant,  and  of 
shifting  the  focus  of  the  study  to  a  more  contextual  and  processual  one, 
rather  than  conclusory. 

B.   IMPLEMENTATION  PROCESS  OUTCOMES 

There  were  three  principal  outcomes  for  the  implementation  phase.  The  first 
concerns  the  statistical  utility  of  the  analyses,  the  second  concerns  auto- 
mation of  the  data,  and,  the  third,  the  focus  of  the  evaluation  reports  them- 
selves. 

Statistical  utility  depends,  for  these  reports,  very  much  on  where  one  sits. 
In  terms  of  strict  research,  none  of  the  analyses  performed  are  truly  analyti- 
cal, past  the  basic  descriptive  mode.  The  statistics  used  were  of  an  excep- 
tionally basic  nature,  and  because  of  missing  data  and  shifted  priorities, 
most  of  the  second  level  analyses  planned  in  the  evaluation  design  could  not 
be  carried  out.  In  terms  of  application,  however,  the  basic  descriptive  sta- 
tistics used  present,  for  probably  the  first  time,  overviews  of  some  of  the 
grant  programs,  broken  out  by  component  parts.  To  put  it  into  artistic  terms, 
what  these  reports  provide  is  the  preliminary  sketch  for  a  painting:  the  com- 
position and  intent  are  evident,  but  the  fullness  and  richness  of  the  entire 
painting  is  not  yet  visible.  Given  different  questions,  and/or  different  time 
parameters  with  regards  to  collecting  missing  data,  it  is  possible  that  such  a 
painting  might  in  time  be  developed.  But  for  the  immediate  purposes  of  the 
Endowment,  it  is  more  useful  to  have  the  sketch,  as  it  is  at  that  phase  of 
development  that  changes  can  be  made  most  easily. 

There  are  two  additional  statistical  constraints:  the  size  of  the  populations 
analyzed,  and  the  audience  for  the  evaluation  report.  Most  of  the  data  files 
created  for  evaluation  purposes  were  of  a  size  sufficient  for  most  statistical 
analyses.  Most  of  these  data  files  were,  however,  subsequently  divided  into 
smaller  units,  which  rendered  much  statistical  analysis  inappropriate  due  to 
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constraints  of  size.  In  addition,  the  people  at  the  Endowment  who  read  the 
reports  are  not  analysts.  They  are  not  statisticians.  They  are,  for  the  most 
part,  individuals  with  particular  talents  in  the  field  of  the  arts  or  of  arts 
service.  Numbers  and  statistical  analyses  have  much  less  meaning  for  them 
than  narrative  descriptions,  and  qualitative  analyses  are  preferred  almost  to 
the  exclusion  of  quantitative  ones. 

A  second  particular  outcome  stems  from  the  mechanics  of  automation.  None  of 
the  information  was  on  computers  of  any  sort.  Records  are  kept,  and  kept 
well,  in  vertical  files.  This  evaluation  effort  represented  one  of  the  first 
attempts  to  format  and  analyze  the  data.  It  is  clear  that  some  type  of  data 
base  management  package  would  be  of  particular  utility  to  the  individual  pro- 
grams in  terms  of  tracking  the  flow  of  information,  of  monitoring  project  per- 
formance and  of  maintaining  an  institutional  memory  that  is  not  dependent  on 
any  one  individual.  It  creates  a  factor  of  accessibility  and  immediacy  not 
present  with  vertical  files  alone,  and  creates  the  ability  to  retrieve  cri- 
tical information  and  issues  by  and  for  Endowment  staff,  for  the  benefit  of 
students  of  the  arts,  and  for  accountability  to  other  government  entities. 

The  third  and  final  outcome  concerned  the  focus  of  the  reports  generated  for 
the  Endowment.  For  example,  for  Design  Arts  personnel,  the  evaluation  report 
has  enhanced  their  ability  to  respond  to  information  requests  by  clarifying 
the  nature  and  scope  of  tasks  required  to  operate  and  manage  a  design  compe- 
tition. The  presentation  of  a  holistic  overview  of  the  program  for  the  last 
several  years  has  also  greatly  facilitated  ongoing  training  workshops  in  com- 
petition design  and  management.  Feedback  from  Design  Arts  personnel  concern- 
ing the  usefulness  of  the  report  includes  its  application  as  a  cross-reference 
tool  for  a  recent  publication  on  design  competitions.  The  evaluation  is 
credited  with  causing  the  program  staff  to  establish  more  complete  files  for 
those  competitions  already  held,  and  to  establish  particular  criteria  for 
monitoring  ongoing  and/or  future  ones.  In  addition,  summaries  of  the  report 
will  be  provided  in  the  panelists'  packages  for  this  year's  examination  of 
grant  applications.  Design  Arts  personnel  have  not  only  their  own,  inside, 
perspective  of  the  relative  processes  underlying  a  successful  or  unsuccessful 
competition,  but  now  also  have  an  outside,  relatively  unbiased,  point  of  view 
of  the  organization  and  controls  which  create  that  process.   As  part  of  the 
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project  monitoring  function,  the  report  has  provided  the  staff  with  more 
insight  on  what  types  of  questions  need  to  be  asked  at  the  various  phases  of 
the  competition,  and  what  particular  efforts  could  best  be  encouraged. 

A  parallel  concern  also  emerged  with  the  focus  of  the  reports,  and  refers  back 
to  the  overall  statistical  utility  of  these  documents.  As  most  of  the  Endow- 
ment personnel  have  little  or  no  familiarity  with  statistics,  the  writing  of 
reports  which  are  so  strongly  based  in  statistics  must  be  altered  radically  to 
minimize  the  use  of  jargon,  to  present  the  findings  in  standard  English,  and 
to  relate  the  analytical  findings  as  closely  as  possible  with  the  actual  cases 
examined  in  the  grant  files.  It  is  not  so  much  literary  style,  but  rather 
that  the  presentation  of  the  data  must  be  as  informal  as  possible,  with  the 
maximum  use  of  description  of  what  the  various  results  mean,  and  far  less 
emphasis  on  the  results  themselves.  In  essence,  the  higher  levels  of  analysis 
of  the  data  are  generated  in  the  writing  of  these  reports:  synthesizing  the 
data  into  a  useful  format  for  the  non-statistical  reader  forces  the 
development  of  hypotheses  and  trends  in  a  coherent  picture. 
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V.  SUMMARY 


Is  evaluation  feasible  for  the  Endowment  given  the  highly  subjective  and  non- 
quantifiable  nature  of  art? 

Over  a  decade  ago,  the  Endowment  itself  recognized  the  value  of  self-evalua- 
tion as  a  management  and  planning  tool.  It  was  also  recognized  that  evalua- 
tion was  routinely,  yet  informally,  applied  throughout  the  granting  process. 
Through  the  two  referenced  contracts  with  Evaluation  Technologies  Incorporated 
(ETI),  Endowment  programs  participating  in  this  pilot  effort  were  shown  how  to 
formalize  that  effort  for  greater  utility. 

Discussions  with  Endowment  staff  early  in  the  evaluation  design  phase  high- 
lighted the  fact  that  programs  have  identified  many  questions  about  the  ef- 
fects of  their  work,  their  impact  on  the  field,  their  constituents,  and  other 
more  specific  issues  of  importance  to  program  planning,  development,  and  man- 
agement. How  are  we  doing?  What  have  we  learned?  Are  we  being  responsive  to 
or  influencing  changes  in  the  arts  field?  Evaluation  research  can  contribute 
to  internal  learning  and  fostering  the  Endowment's  valuable  public  relation- 
ships. 

This  pilot  evaluation  effort  has  demonstrated  that  by  altering  and  refining 
the  information  management  systems  employed  by  each  program,  through,  for 
example,  the  use  of  more  targeted  information  collection  instruments  and 
automation  of  the  data  files,  these  types  of  questions  can  readily  be  answer- 
ed. The  evaluation  effort  identified  the  existing  and  potential  sources  of 
information,  and  demonstrated  their  usefulness  in  evaluation  and  information 
research. 

Can  program  activities  be  defined  within  an  evaluation  framework  without 
infringing  upon  or  threatening  subjective  and  expert  panel  judgements? 

The  types  of  evaluation  studies  requested  by  the  programs  we  worked  with  did 
not  put  the  Endowment  in  the  position  of  judging  the  quality  of  the  artistic 
endeavors  pursued  by  grantees,  but  rather  measured  quality  in  terms  of  the 
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arts  community's  response  to  the  grant  project.  The  evaluation  designs  also 
examined  the  grantees'  abilities  to  effectively  manage  the  project  and  the 
grant  funds.  As  mentioned  earlier,  Endowment  goals  regarding  such  issues  as 
"innovativeness"  and  work  of  "highest  quality"  were  easily  interpreted  within 
the  context  of  program  category  activities. 

In  fact,  program  staff  members  found  the  task  of  clearly  defining  and  matching 
program/category  goals  and  the  information  required  to  assess  goal  achievement 
a  yery  useful  exercise.  We  suggest  that  all  programs  undertake  this  type  of 
activity  whether  or  not  a  full  evaluation  is  to  be  performed.  It  enhances  the 
program's  understanding  of  what  information  needs  to  be  collected  from  appli- 
cants and  grantees,  and  for  what  purpose(s).  Grant  application  forms,  supple- 
mental information  sheets,  site  visit  records,  and  interim  and  final  descrip- 
tive and  financial  report  requirements  can  then  be  revised  with  the  knowledge 
that  only  necessary  information  is  being  collected,  while  ensuring  that  criti- 
cal information  is  not  left  uncaptured.  These  activities  will  serve  to  reduce 
the  reporting  burden  of  grantees  and  the  information  management  activities  of 
program  staff.  We  further  recommend  that  the  same  forms  be  used  before  and 
after  the  grant  project  is  conducted  to  allow  direct  comparisons  of  planned 
and  actual  activities  and  expenses. 

Can  evaluation  activities  be  performed  by  Endowment  staff  within  the  context 
of  their  current  grant-cycle  responsibilities? 

The  initial  evaluation  design  and  implementation  tasks  performed  by  ETI  were 
unusually  time  consuming  due  to  a  number  of  anticipated  factors  which  relate 
to  the  start-up  of  any  new  project,  including  the  initial  participant  learning 
curve  and  the  establishment  of  working  relationships.  More  specifically,  in 
this  case,  efforts  to  reconcile  years-old  grant  records  and  frequent  changes 
in  grantee  reporting  requirements  with  the  information  requirements  of  the 
evaluation  design  also  created  some  delays  in  the  evaluation  implementation 
phase. 

Program  personnel  generally  possess  backgrounds  and  expertise  in  the  arts,  not 
in  management  sciences  or  research  methodologies.  However,  ETI  consistently 
found  Endowment  staff  members  receptive  to  and  interested  in  expanding  their 
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capabilities  in  these  disciplines,  although  at  times  appeared  somewhat  intimi- 
dated by  their  new  skills.  It  is  clearly  evident  that  all  Endowment  staff 
could,  with  minimal  coaching,  perform  evaluation  design  and  implementation 
functions  at  the  same  level  at  which  ETI  has  performed. 

Furthermore,  once  the  historical  data  bases  have  been  compiled,  as  they  now 
have  in  Design  Arts  and  Inter-Arts,  the  task  of  maintaining  them  can  become 
routine,  and  will  even  decrease  the  amount  of  time  it  currently  takes  for 
program  specialists  to  file,  maintain,  and  retrieve  specific  records.  Their 
ability  to  respond  to  panelist  and  grant  applicant  inquiries  will  also  be 
greatly  facilitated. 

What  is  missing  is  the  incentive  for  programs  to  change  their  approach  to  in- 
formation management  --  the  personal  interest  among  program  specialists  is 
there  and  the  capability  to  automate  grant  records  exists,  but  there  is  no 
mandate  or  management  initiative  to  do  so.  It  is  therefore  recommended  that 
an  in-house  evaluation  technical  assistance  capability  be  established,  or,  at 
the  y/ery  least,  a  how-to  manual  be  prepared  for  Endowment-wide  distribution. 
Additionally,  training  on  evaluation  methods  should  be  offered  for  program 
personnel. 

Can  evaluation  results  be  a  useful  tool  for  managers  and  panelists? 

As  program  budgets  are  curtailed,  the  importance  of  truly  evaluating  program 
performance  is  heightened.  Emphasis  should  be  placed  on  identifying  areas 
where  cuts  can  be  made  while  maintaining  optimum  program  effectiveness. 

In  a  more  narrow  focus,  evaluation  outcomes  can  be  used  for  program-specific 
purposes,  including: 

o  Definition  of  information  requirements 

o  Identification  of  grantee-specific  accomplishments  and  problems 
for  consideration  by  panels  tasked  with  making  funding 
recommendations 
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o  Program  planning,  policy  making,  and  advocacy 

o  Preparation  of  a  lessons  learned  compendium 

o  Preparation  of  best  practices  handbooks  and  seminars  for  grantees 
and  other  constituents. 

As  outlined  in  Chapters  III  and  IV,  it  is  recommended  that  the  Endowment  ex- 
pand its  use  of  evaluation  techniques,  and,  perhaps  more  importantly,  consider 
the  implementation  of  more  targeted  and  less  time  consuming  information  col- 
lection and  management  systems  within  each  program  office. 
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